Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Tag ignored tests that require SubqueryBroadcastExec #647

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

parthchandra
Copy link
Contributor

Which issue does this PR close?

Addresses part of #551

Rationale for this change

This addresses the following tests which require SubqueyBroadcastExec to be implemented

  • partition pruning in broadcast hash joins with aliases (appears twice)
  • partition pruning in broadcast hash joins (appears twice)
  • different broadcast subqueries with identical children
  • SPARK-32817: DPP throws error when the broadcast side is empty
  • SPARK-36444: Remove OptimizeSubqueries from batch of PartitionPruning
  • SPARK-38674: Remove useless deduplicate in SubqueryBroadcastExec
  • SPARK-39338: Remove dynamic pruning subquery if pruningKey's references is empty
  • SPARK-39217: Makes DPP support the pruning side has Union
  • join with ordering requirement

@parthchandra
Copy link
Contributor Author

@kazuyukitanimura please take a look

@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 53.81%. Comparing base (eff2897) to head (bed0db9).
Report is 5 commits behind head on main.

Additional details and impacted files
@@              Coverage Diff              @@
##               main     #647       +/-   ##
=============================================
+ Coverage     33.42%   53.81%   +20.39%     
- Complexity      805      811        +6     
=============================================
  Files           109      106        -3     
  Lines         42462    10245    -32217     
  Branches       9342     1917     -7425     
=============================================
- Hits          14191     5513     -8678     
+ Misses        25322     3755    -21567     
+ Partials       2949      977     -1972     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@@ -442,7 +442,7 @@ index 2c24cc7d570..50a2ce86117 100644

- test("partition pruning in broadcast hash joins with aliases") {
+ test("partition pruning in broadcast hash joins with aliases",
+ IgnoreComet("TODO: https://github.com/apache/datafusion-comet/issues/551")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, aren't they already ignored? Seems this patch only changes the ignore reason?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This 4.0.0 diff was created before #437 was merged, so these tests are not specifically related to Spark 4.0 issues that is tracked in #551

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant before this PR, the diffs already have IgnoreComet which ignores these tests. Isn't?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is correct, these tests are ignored currently, so no change in test coverage. The purpose of this PR is to tag them with #242 so that they will be fixed all together.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, thanks for confirming it. The PR title seems inaccurate so.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point @parthchandra Would you like to update the title?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the commit and PR title.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @parthchandra

Copy link
Contributor

@kazuyukitanimura kazuyukitanimura left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@parthchandra parthchandra changed the title fix: Disable tests that require SubqueryBroadcastExec fix: Tag ignored tests that require SubqueryBroadcastExec Jul 10, 2024
@viirya viirya merged commit fdd2c4f into apache:main Jul 10, 2024
73 checks passed
@viirya
Copy link
Member

viirya commented Jul 10, 2024

Merged. Thanks @parthchandra @kazuyukitanimura

himadripal pushed a commit to himadripal/datafusion-comet that referenced this pull request Sep 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants